- name: d8.control-plane.etcd.availability
  rules:

  - alert: KubeEtcdTargetDown
    expr: max by (job) (up{job="kube-etcd3"} == 0)
    for: 1m
    labels:
      severity_level: "5"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_ignore_labels: "job"
      description: >
        Check the status of the etcd Pods: `kubectl -n kube-system get pod -l component=etcd`
        or Prometheus logs: `kubectl -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus`.
      summary: Prometheus is unable to scrape etcd metrics.

  - alert: KubeEtcdTargetAbsent
    expr: absent(up{job="kube-etcd3"}) == 1
    for: 1m
    labels:
      severity_level: "5"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_ignore_labels: "job"
      plk_create_group_if_not_exists__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: >
        Check the status of the etcd Pods: `kubectl -n kube-system get pod -l component=etcd`
        or Prometheus logs: `kubectl -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus`
      summary: There is no etcd target in Prometheus.

  - alert: KubeEtcdNoLeader
    expr: max by (node) (etcd_server_has_leader{job="kube-etcd3"}) == 0
    for: 1m
    labels:
      severity_level: "4"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_unavailable: "KubeEtcdUnavailable,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: >
        Check the status of the etcd Pods: `kubectl -n kube-system get pod -l component=etcd | grep {{ $labels.node }}`.
      summary: The etcd cluster member running on the {{ $labels.node }} Node has lost the leader.

- name: d8.control-plane.etcd.malfunctioning
  rules:

  - alert: KubeEtcdHighNumberOfLeaderChanges
    expr: max by (node) (increase(etcd_server_leader_changes_seen_total{job="kube-etcd3"}[10m]) > 3)
    labels:
      severity_level: "5"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_caused_by__ping: "NodePingPacketLoss,tier=cluster,prometheus=deckhouse,destination_node={{ $labels.node }},kubernetes=~kubernetes"
      plk_create_group_if_not_exists__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: |
        There were {{ $value }} leader re-elections for the etcd cluster member running on the {{ $labels.node }} Node in the last 10 minutes.

        Possible causes:
        1. High latency of the disk where the etcd data is located;
        2. High CPU usage on the Node;
        3. Degradation of network connectivity between cluster members in the multi-master mode.
      summary: The etcd cluster re-elects the leader too often.

  - alert: KubeEtcdInsufficientMembers
    expr: count(up{job="kube-etcd3"} == 0) > (count(up{job="kube-etcd3"}) / 2 - 1)
    for: 3m
    labels:
      severity_level: "4"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: >
        Check the status of the etcd pods: `kubectl -n kube-system get pod -l component=etcd`.
      summary: There are insufficient members in the etcd cluster; the cluster will fail if one of the remaining members will become unavailable.

  - alert: KubeEtcdHighFsyncDurations
    expr: max by (node) (histogram_quantile(0.99, rate(etcd_disk_wal_fsync_duration_seconds_bucket{job="kube-etcd3"}[5m])) > 0.5)
    for: 10m
    labels:
      severity_level: "7"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: "markdown"
      plk_create_group_if_not_exists__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      plk_grouped_by__kube_etcd_malfunctioning: "KubeEtcdMalfunctioning,tier=cluster,prometheus=deckhouse,kubernetes=~kubernetes"
      description: |
        In the last 15 minutes, the 99th percentile of the fsync duration for WAL files is longer than 0.5 seconds: {{ $value }}.

        Possible causes:
        1. High latency of the disk where the etcd data is located;
        2. High CPU usage on the Node.
      summary: Synching (fsync) WAL files to disk is slow.
